LPT Orsay 11-38 
TTP 11-13 



Reduced branching ratio for H — > AA — > 4 r 
from A — r]i) mixing 

Florian Domingo^ and Ulrich Ellwanger^ 

^ Institut fiir Theoretische Teilchenphysik, KIT, Universitat Karlsruhe, 
D-76128 Karlsruhe, Germany 
Laboratoire de Physique Theorique, UMR 8627, CNRS and Universite de Paris-Sud, 

Bat. 210, F-91405 Orsay, France 



Abstract 

Models with an extended Higgs sector, as the NMSSM, allow for scenarios where 
the Standard Model-like CP-even Higgs boson H decays dominantly as H ^ AA — )• 
4t where A is a light CP-odd Higgs boson. Tight constraints on this scenario in the 
form of lower bounds on Mh have recently been published by the ALEPH group. We 
show that, due to A — rji, mixing, the branching ratio H — )• AA — t- 4r is strongly 
reduced for in the range 9 — 10.5 GeV. This is the range of in which the 
tension between the observed r]h{lS) mass and its prediction based on QCD can be 
resolved due to mixing, and which is thus still consistent with a light CP-even Higgs 
boson H satisfying LEP constraints with a mass well below 114 GeV. This result is 
practically independent from the coupling of ^ to 6 quarks. 



1 Introduction 



One of the main goals of the Large Hadron Colhder (LHC) is the detection of the Higgs 
boson, or of at least one of several Higgs bosons if corresponding extensions of the Standard 
Model (SM) are realized in nature. These searches depend crucially on the Higgs production 
cross sections and the Higgs decays. 

In the case of the SM, the production cross sections and decay branching ratios are quite 
well known as functions of the still unknown Higgs mass [1]. In the Minimal Supersymmetric 
Standard Model (MSSM) with its extended Higgs sector, these quantities have been studied 
as well and it seems that at least one of the Higgs bosons cannot be missed at the LHC [2]. 
There exist, however, well motivated scenarios with somewhat more extended Higgs sectors, 
as the Next-to-Minimal Supersymmetric Standard Model (NMSSM, see [3,4] for recent 
reviews), where the Higgs decays can differ strongly from both the SM and the MSSM. 
It is very important to be aware of the possibility of such unconventional Higgs decays; 
the absence of a signal in standard Higgs search channels may otherwise be completely 
misinterpreted. 

The Higgs sector of the NMSSM consists of two SU(2) doublets Hu and Hd (as in 
the MSSM), and one additional gauge singlet S. Due to its coupling XSHuHd in the 
superpotential, a vacuum expectation value (vev) s of 5* generates a supersymmetric mass 
term /igg = As for Hu and Hd- Since s and hence fies are naturally of the order of the 
soft Susy breaking terms ~ Msusy, this solves the so-called /i-problem of the MSSM [5]. 
(This remains true in the limit A, k — 0, where k, is the singlet self-coupling in the 
superpotential, leading to s ~ Msnsy/i^, but fics ~ (A//t)Msusy ~ ^susy) Furthermore, in 
its simplest Z3 invariant version, the superpotential of the NMSSM is scale invariant; it 
is in fact the simplest phenomenologically acceptable supersymmetric extension of the SM 
with this property. 

The physical neutral Higgs sector in the NMSSM consists of 3 CP even and 2 CP odd 
states. (Here we do not consider the possibility of CP violation in the Higgs sector.) In 
general, these states are mixtures of the corresponding CP even or CP odd components of 
Hu, Hd and S, without the CP odd Goldstone boson swallowed by the massive Z boson. 
Often, one of the CP even states is SM like, i.e. with similar couplings to gauge bosons as 
the SM Higgs boson (but with possibly enhanced couplings to quarks and leptons), with a 
mass bounded from above by ~ 140 GeV [6]. At first sight, the detection at the LHC of this 
Higgs boson - denoted subsequently by H for simplicity - seems to be guaranteed, given 
the lower LEP bound of ~ 114 GeV on masses of Higgs bosons with SM like couplings to 
the Z boson and SM like decays. 

However, the lighter of the two CP odd states (denoted by Ai) could have a mass Ma^ 
below half of the mass of H [7,8]. Then, H would decay dominantly as H ^ AiAi, 
since this coupling is typically larger than the coupling of H to b quarks [7,8]. Such a decay 
of H would have important consequences both for lower bounds on its mass from searches 
at LEP, and for its detection at the LHC. Now the H final decay products depend on Ma^'- 
for Mai > 10.5 GeV, they consist mainly of 4 b quarks (with some 26 + 2r admixture), 
whereas for 3.5 GeV < Ma^ < 10.5 GeV, they consist mainly of 4 r leptons (with some 
small 2t + 2 n admixture). In fact, H 4b decays have also been searched for by OPAL 
and DELPHI at LEP [9, 10] implying > 110 GeV if H has SM like couphngs to the Z 
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boson [11]. On the other hand, LEP constraints on i7 — )■ 4r decays were relatively weak, 
allowing for Mh as low as ~ 90 GeV [11]. 

This led to the scenario advocated in [13-16] (see also [17]) with Mh ^ HO GeV, 
Ma^ < 10.5 GeV, a dominant (but not exclusive) decay H — )■ AiAi — )■ 4 leptons and a low 
finetuning among the soft Susy breaking parameters due to the relatively low mass of H. A 
remaining small branching ratio for iJ — t- 2 6 could explain the 2 a excess observed in this 
channel for Mh ~ 100 GeV [11, 14]. 

The final state H ^ At has recently been reanalysed by the ALEPH group [12] implying 

upper bounds on = jJ^lXe^^zH) ^ ^^(^ ^ 2Ai) x BR{Ai t+ r'f as function 
of Mh and Ma^- These bounds seem to impose strong constraints on the above scenario, 
unless a{e^e~ ZH) and/or the BR{H 2 Ai) and/or the BR[Ai t^) are smaller 

than naively expected [18]. 

A light CP odd scalar Ai would also have important consequences for the physics of hh 
bound states. These effects depend on the coupling of Ai to h quarks. Normalized relative 
to the coupling of the SM Higgs boson, the coupling of Ai to b quarks is given by Xd with 

= cos 6'a tan /3 , (1) 

where cos 6*^1 denotes the SU(2) doublet component of Ai, and tan/3 is the usual ratio of 
Higgs vevs Vu/vd- For tan/3 much larger than 1, Xd could satisfy Xd 3> 1 as well. {Xd is 
simultaneously the coupling of Ai to leptons normalized relative to the coupling of the SM 
Higgs boson.) 

In fact the relation (1) is valid for Ai in any extension of the SM with two Higgs 
doublets Hu (coupling exclusively to up-type quarks) and Hd (coupling exclusively to down- 
type quarks and leptons), but arbitrary singlets. Our subsequent results depend only on 
Ma^ and Xd-, and are valid for any such models. In the NMSSM, a light CP odd scalar 
Ai can play the role of a pseudo Goldstone boson of an approximate R- or Peccei-Quinn 
symmetry [7,8]. Then, however, one always has cos^a ~ 1/ tan/3 [4] and hence Xd < 1. 

Since the pseudoscalar hb bound states T]h{nS) have the same quantum numbers as a CP 
odd Higgs Ai, the states rih{nS) and Ai can mix [19-22] with important consequences both 
for the mass spectrum and the decays of the physical eigenstates. A state rj^^lS) has been 
observed in radiative T{3S) and T(2S') decays by BABAR [23,24], with the result that its 
mass of 9390.9 ±3.1 MeV is below the one expected from most QCD predictions for the 
T(15') — ijbi^S) hyperfine splitting [25-27]. Indeed, such a mass shift could be explained 
by the mixing of 7]iy{lS) with Ai provided Mai (before mixing) is in the 9.4 — 10.5 GeV 
range [22]. 

On the other hand, Ai can be searched for in radiative decays T{nS) — ?■ 'yAi, Ai — )■ 
2 leptons. (See [28] for a discussion of 1]^ -> r+r~ mediated by Ai.) Unsuccessful searches 
by CLEO [29] and BABAR [30,31] lead to upper bounds on Xd as function of Ma^, which 
have been studied in [18,21,32] for Ma^ < 9 GeV where the r]i,{nS) — Ai mixing is not very 
relevant. Notably for Ma^ below the 2r threshold, where Ai has a large branching fraction 
into two muons, these bounds are quite strong and imply Xd < 0.5. Upper bounds on Xd 
for 8 GeV < Ma^ < 10.1 GeV, including effects from rib{nS) — Ai mixing, have recently 
been investigated in [33], implying Xd < 2... 7 depending on Ma^. (These bounds are 
consistent with limits from the violation of lepton universality in inclusive T{nS) decays as 
proposed in [20,34-36] and studied in [37,38].) 
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Possible riii{nS) — Ai mixings would also affect the ALEPH bounds on if — 2 — )■ 4 r 
[12] in the interesting mass range 9 GeV < Ma^ < 10.5 GeV, since Ai decaying hadron- 
ically through its rji, components would imply a different signature. The corresponding 
consequences for this process have not been taken into account before; this study is the 
purpose of the present paper. In fact, our result is quite dramatic: the ALEPH bounds 
imply practically no constraint on the BR{H — ?■ 2 Ai) in the corresponding mass range, 
since the BR[Ai — ?■ r+r~) tends to be very small even for small values of X^. The origin of 
this phenomenon can easily be understood qualitatively: the width of the decay Ai — > r+r~ 
of the pure state Ai, albeit proportional to Xj (which appears also in the coupling of Ai 
to r leptons), is always much smaller than the hadronic width of the rjbinS) to hadrons 
given the present upper bounds on X^- Hence, even a small admixture of T]h{nS) to any 
physical eigenstate implies a large hadronic decay width, suppressing the branching ratio 
of the physical state into r+r" and making it very difficult to detect. For X^ < 10 this 
effect is approximately independent from Xd, since both the width for Ai — )■ t~^t~ and the 
rib{nS) — Ai mixing are proportional to Xj. 

In the next Section we study this phenomenon quantitatively, with the result stated 
above. In Section 3 we briefly comment on the impact of our result on future Higgs searches. 



2 The BR[H — > 4t) in the presence of A — 775 mixing 

In this section we consider the mixing of a CP odd Higgs state Ai (denoted by A for 
simphcity) with the states r]b{lS), Vb{2S) and r]b{3S) with masses below the BB threshold. 
The mass squared matrix in the basis rjb{lS) — rjbi^S) — rib{3S) — A can be written as [22] 
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The off-diagonal elements depend on the rib{nS) wave functions at the origin, and X^ 
as given in (1) multiplied by the coupling of a SM like Higgs boson to h quarks [19-22]. 
Estimating the wave functions at the origin as in [20-22] one obtains 
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0.14 GeV^ X Xd 
0.11 GeV^ X Xd 
0.10 GeV^ X Xd 



(3) 



The errors on these quantities are about 10%, but our subsequent results are not sensitive 
to the precise numerical values. For the diagonal elements To^^f^^g) we take [25] m.^^^[2S) = 
10002 MeV, m^^i^ss) = 10343 MeV. "'^.^^(15) is determined, for given Ma and Xd, by the 
condition that the state with its mass of ~ 9391 MeV observed in radiative T(35') and 
T(25') decays by BABAR [23,24] must be identified with one of the eigenstates of M.'^. 
Again, our subsequent results depend only weakly on these masses. 
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It is straightforward to diagonalize the mass matrix (2). The 4 eigenstates will be 
denoted by 77^, which are decomposed into the unmixed states as 

r]i = P,,i rjbilS) + Pi,2 Vb{2S) + Pi-s Vb{^S) + P,,4 A . (4) 

Both the eigenvalues of the mass matrix (2) and the mixing coefficients Pij in (4) depend 
on the unknown mass M^. Let us recall some obvious properties of the eigenvalues and 
the mixing coefficients: whenever is far from any of the m^j^(„5), the mixing will be 
relatively small (but increasing with Xd), and A will be an approximate mass eigenstate. 
For fixed Xd, the closer Ma is to nirjf^^ns), the larger the A — rib{nS) mixing will be, resulting 
in shifts of the eigenvalues of Ai^ w.r.t. its diagonal elements. 

We recall that the state with a mass of ~ 9391 MeV observed by BABAR must be 
identified with one of the eigenstates of Ai^. Independently from the value of the diagonal 
element m^j^(i5) of Ai^, it follows that cannot be arbitrarily close to 9391 MeV unless 
the mixing (and hence Xd) tends to zero. This consideration leads to an upper bound on Xd 
depending on Ma, with Xd ^ for Ma ^ 9391 MeV, and still Xd < 20 for Ma ~ 10 GeV 
or Ma ~ 8.5 GeV [21]. 

Next we turn to the decays of the eigenstates rji, starting with the decays of the states 
before mixing. A will decay dominantly into A ^ t~ , with a partial width given by 

= X f^'^l'^^ ll-A^ ~ Xj X 1.9 . 10- MeV x ( . (5) 

We determine the BR{A r+r") from NMHDECAY [39,40] inside NMSSMTools [41] 
(assuming tan/3 ~ 5), which gives BR{A — r+ r^) ~ 0.9 — 0.75 with increasing Ma, the 
remaining BR originating from A decays into cc quarks and gluons. (A smaller BR{A — >■ 

T~), as advocated for some parameter choices in [18], would only amplify our subsequent 
conclusions.) Hence we take F^'* ~ (1.1 — 1.33) x F^^. 

The states rji,{nS) (before mixing) would decay nearly exclusively into hadrons (like the 
states ric{nS)). Using the formalism in [42], the widths of the states rib{nS) can be estimated 
from the widths of the corresponding T states and the rib{nS) masses. Subsequently we take 
Tn,(is) = 11.8 MeV, F^,(25) = 5.4 MeV and Tn,(3S) = 3.9 MeV. Note that, unless Xd > 10, 
these widths are much larger than F^"^. (We recall that, for Ma < 10.1 GeV, Xd < 2 ... 7 
due to constraints from T{nS) — >■ jAi, Ai — )■ 2 leptons [33].) 

In terms of these widths and the mixing coefficients, the BR{rii t^) of the eigen- 

states rji are given by [22] 

BR{r^, ^ r+ r') = -— . (g) 

( Pi,n^Vb{nS)j + PI,J''a 

Let us consider the state r]i with the largest A component, i.e. the largest coefficient 
Pj\. (Since, essentially, A mixes with just one of the 7]b{nS) states depending on Ma, there 
exists always one state with Pf^ > 0.5.) Its PP(r/i — >■ r"*" r ) is smaller than 0.9 — 0.75 due 
to the terms ~ ^^^(^nS) in the denominator of (6). In fact, even if Pj^„ <^ 1, these terms are 
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often numerically dominant due to r^^„5) ^ F^*, implying a considerable reduction of the 
BRijii r+ r~). For < 5, the result is nearly independent from X^, since F^'^ and F^* 
as well as Pf^ are proportional to Xj, and cancels out. 

In Fig. 1 we show the BR{r]i — t~) for the state rji with the largest A component as 
function of Ma for Xd = 1. Depending on Ma, this state corresponds to ?7i . . . 774, which is 
indicated by the various colors. For X^, = 1, the mass of this state is practically identical to 
Ma- Usually, the branching ratios into r'^ t~ of the remaining states are neglibibly small. 
Note that, whenever Ma is close to any of the masses mrj^^ris)^ the mixing becomes strong 
(i^^4 ~ P^^ ~ 1/2) leading to BR{rii r+r") ~ F^'"/F^^(„5), which is very small. (As 
stated above, we must have Xd for Ma — 9391 MeV. This upper bound is applied 
to Xd for Ma ~ 9391 MeV in Fig. 1, but = 1 is used for all other values of Ma-) 
Remarkably, even if Ma is not close to any of the masses mj^^(^nS)i the suppression of the 
BRijii T~) is still quite strong due to the terms ~ F^j^(„5) in the denominator of (6), 

and BR{rii — )■ r"*" r~) < 0.65 for any Ma in the range 9 — 10.5 GeV. 

ll 




Ma 

Figure 1: The BR{rii — )■ t~) for the state r/j with the largest A component as function 
of Ma for Xd = 1- The colors indicate which state 77^ is concerned (red^ r/i, green—)- 772, 
brown—)- 773, blue—)- 7/4). 

Finally we have to re-interpret the decay H — )■ A A — )■ 4 r in the presence ol A — irii,{nS) 
mixing: now this process corresponds to X]^j=i(-^ ~^ ViVj — ?■ 4r). The coupling of the 
states 7/j to H (originating from the coupling of A to H) is proportional to Pj^4, and we can 
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write 



^BR{H riifij ^ 4r) = BR{H ^ AA) x R 



R 



i=l 



(7) 



We can compute R as function of Ma and Xd, and the result is shown in Fig. 2. 
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Figure 2: The function R, defined in (7), in the plane vs. Ma- Also indicated are 
upper bounds on Xd from CLEO (red), BABAR (blue) and from the condition that one 
eigenstate of the mass matrix (2) has a mass of 9391 MeV with rrir^^iis) within a reasonable 
range (green). 



In Fig. 2 we also show upper bounds on Xd from CLEO (red), BABAR (blue) and 
from the condition that one eigenstate of the mass matrix (2) has a mass of 9391 MeV and 
m^^(is) (before mixing) is within a range 9360 — 9445 MeV covered by QCD predictions 
(green). Hence, for Ma < 10.1 GeV, only small values of Xd-, where R is nearly independent 
from Xd (as explained above), are of interest. Like the BR{rji — )■ r~), R varies strongly 
with Ma- It follows from R ~ Y/.=i PiA ^ BRiVi ^ ^'f that R never exceeds 0.4 for 
Ma in the range 9 — 10.5 GeV, and R ~ (T'^/ /T.fj^(^nS)Y (which is tiny) as soon as Ma is 
near any of the masses mj^^(^ns)- Now the quantity constrained by ALEPH (see Fig. 6 
in [12]) must be interpreted as = ^'2 ^ ^ aj^+e-^^m^^ ^ 5i?(^r ^2 A). It follows 

that is left unconstrained at least for Mh > 98 GeV and Ma in the range 9 — 10.5 GeV, 
as well as for any lower value of as long as Ma is in the range where R in Fig. 2 
is below 0.2, corresponding essentially to a BR{rii — r"*" r~) in Fig. 1 below ~ 0.5 (but 
depending slightly on Xd)- Since, in addition, one always has ^'^ < 1 even if the process 
if —J- 2/1 is kinematically allowed (since the BR[H bb) is never exactly zero), scenarios 
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with Mh < 98 GeV are consistent with the ALEPH constraints as well for most values of 
Ma in the range 9 - 10.5 GeV. 

3 Conclusions and outlook 

After the publication of the ALEPH analysis [12] it seemed that the attractive scenario with 
a light CP-even Higgs boson H and a mass Mh well below 114 GeV, decaying dominantly 
as — 7- 2 A — 7- 4 r, was tightly constrained. We have shown that these constraints are 
absent for Mh > 98 GeV and Ma in the range 9— 10.5 GeV, and in the case of lower values 
of Mh for most values of Ma in this range. The origin is a reduced BR{A — )■ r"*" r~) caused 
hy A — rib{nS) mixing, leading to dominant hadronic decays of the physical eigenstates. 
This window for Ma is of particular interest, since it contains the region in which the 
tension between the observed ?7b(lS') mass and its prediction based on QCD can be resolved 
[22,33] through this mixing. We emphasize that we did not make particular assumptions 
on the SU(2) doublet component cos6'^, on tan/3 or on the coupling Xd (see (1)) of A to 
h quarks since, at least for small mixing angles, cancels out in the expression (6) for the 
BRijii — T'^ T~) for the mass eigenstates. 

For small Xd and a correspondingly small A — ri[,[nS) mixing, this result seems counter- 
intuitive at first sight. However, the point is that already a small admixture of any rjblnS) 
state to the mass eigenstate rji suffices such that the mass eigenstate rji decays dominantly 
hadronically, since the corresponding hadronic widths of rii,{nS) are much larger than P^^. 
This remains true for small Xd, since then P^^ becomes small as well. 

The consequences of this scenario for Higgs searches at the LHC would be quite dramatic, 
since the dominant Higgs decay mode would he H ^ 2 A ^ hadrons and, like in the 
scenarios discussed in [43-45], the H signal would be buried under the QCD background. 
Moreover, dominant hadronic decays of the mass eigenstate rji would also handicap searches 
for A via central exclusive production [46] at hadron colliders, or via the fi~^ fi~ final state 
as proposed in [47] and studied, using early LHC data, in [48] . It remains to look for A in 
radiative T decays, but corresponding searches have also to be interpreted carefully taking 
mixing effects into account [19-21,33]. 
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